Appropriately Timed AHA Statement on Methodological Standards on Systematic Reviews and Meta-Analysis on Prevention and Treatment Studies

Evidence-based medicine requires the interpretation of best available evidence and its integration with patient’s values and physician’s judgment.¹ In this regard, meta-analyses are considered by many at the top of the pyramid of evidence-based medicine. Meta-analyses combine and summarize data from many studies effectively increasing the sample size and power to detect an effect. When done properly, meta-analyses have the potential of leading to lifesaving resolution of misconceptions. For example, many underpowered trials did no show benefit in treating woman at risk of pre-term birth with corticosteroids. A meta-analysis of such trials, however, showed pre-natal steroids to be lifesaving for the newborns at risk of dying due to immaturity.^2,3

Considering all meta-analyses, due to their study design, as best quality of the evidence is insufficient and an overly simplistic approach. Meta-analyses are at risk of bias if they are not conducted with methodological rigor.

The Methodological Standards for Meta-Analyses & Qualitative Systematic Reviews of Cardiac Prevention and Treatment Studies statement by the American Heart Association (AHA) effectively summarizes key steps required for initiation and acceptable completion of such reviews.⁴ Rao and colleagues outline key methodological components of systematic reviews and meta-analyses, describing possible methodological options, and their rationale, necessary for successful planning and conduct of such studies. The discussions contained within each section are adequately broad and thus may be applicable to other fields beyond cardiac prevention and treatment. This statement represents a step forward to enhance quality of meta-analysis to continue fulfilling their purpose of adequately summarizing the available best quality evidence.

The AHA statement recommends pre-planning the methods to be used in conducting a systematic review and meta-analysis. Different strategies are to publish protocol of meta-analysis or registration of protocols on PROSPERO. The latter has a number of merits including but not limited to outlining the different steps required for conducting such reviews and keep a historic record of the original plan; however, it suffers from two main limitations. If authors are unsure of their decisions and plans for carrying out a proposed systematic review and meta-analysis, publication of a protocol on PROSPERO,⁵ will not provide peer reviewed feedback to improve on the proposed plan. Publication of a protocol in a peer reviewed journal (such as BioMed Central, Systematic Reviews, Evidence-based Preclinical Medicine) may overcome this limitation by receiving adequate and helpful feedback to refine the protocol and steps for carrying out the review.

Another limitation of PROSPERO is its lack of endorsement for overall quality of evidence. As outline in the AHA statement, the approach to assessing the overall quality of the evidence should also be systematic and defined a priori. More increasingly, guideline developers rely on systematic reviews and meta-analyses as the highest form of evidence when making various forms of recommendations⁶. For panelists or guideline developers and general evidence users, the overall quality of the evidence informing the credibility of the pooled effect estimate is fundamental.

Certain limitations related to the synthesis of the available evidence including risk of bias or methodological flaws in studies included in a meta-analysis, impression in effect estimates, heterogeneity or variation in results across included studies, indirectness or differences in population, outcomes or intervention used, or publication bias, may affect the quality of the evidence. These factors should be assessed in combination to adequately interpret the evidence. The current statement by AHA addressed the role of the Grades of Recommendation, Assessment, Development, and Evaluation (GRADE) instrument in judging the overall quality of the evidence. This tool, as outlined in the statement, is endorsed by over ninety organizations.⁷ The GRADE tool categorizes the overall quality of the available body of evidence for each outcome as very low, low, moderate, and high. The GRADE rating system facilitates the process of evidence appraisal for decision making and for use in health care recommendations in clinically critical and important outcomes.

One of the main aspects when conducting a systematic review and meta-analysis is to evaluate consistency in results.⁸ The presence of inconsistency or heterogeneity in the available evidence can be seen as a two-faced scenario: a limitation or an opportunity for new knowledge.

One face represents a limitation in performing a meta-analysis. The presence of substantial differences in methodology (i.e. randomized controlled trial and case-control studies), population (i.e. pediatrics and adults), intervention (medical and surgical therapy) or comparison (placebo and active control therapy) or outcome (all-cause and specific-cause mortality) may preclude pooling or the performance of a meta-analysis, as the AHA statement recommends. The presence of methodological heterogeneity as assessed by statistical test such as I2 should not be used to decide on whether one should pool or not. In fact, the decision of pooling should also be specified a priori and at this stage the presence of statistical heterogeneity is unknown or not yet evaluated.

The other face related to the presence of inconsistency represents an opportunity to generate new knowledge or confirm pre-specified hypotheses. The presence of heterogeneity in results of studies with different characteristics permits systematic reviews and meta-analysis to propose and identify factors associated with differences in effect estimates which could be further corroborate in subsequent studies. The AHA statements offers substantial guidance on how to approach the evaluation of heterogeneity in meta-analysis to ensure truthful results and conclusions.

The AHA statement provides a compiled guide on special circumstances, such as adequacy of pooling, choice of statistical methods for pooling, dealing with no events in individual studies, methods for sensitivity and subgroup analysis and meta-regression, assessment of publication bias and understanding of funnel plots, network meta-analyses, Bayesian methods, and meta-analyses of individual patient data.

In the current era of medicine, now more than ever before, practice changing evidence, primarily in the form of randomized control trials, is being published and disseminated to all users. Clinicians caring for patients impacted by such evidence will benefit from rapid and trustworthy creation or update of clinical practice guidelines.⁶ Highest quality and strongest form of evidence should inform these recommendations. It is thus of great importance to ensure future authors of systematic reviews and meta-analyses are aware of necessary steps and decisions required for planning and producing high quality work, which is further support for this appropriately timed AHA statement.